UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

September 10, 2004 



THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COPY FROM 
THE RECORDS OF THE UNITED STATES PATENT AND TRADEMARK 
OFFICE OF THOSE PAPERS OF THE BELOW IDENTIFIED PATENT 
APPLICATION THAT MET THE REQUIREMENTS TO BE GRANTED A 
FILING DATE. 



APPLICATION NUMBER: 60/496,426 
FILING DATE: August 20, 2003 

RELATED PCT APPLICATION NUMBER: PCT/US04/25366 



Certified by 




iJon W Dudas 



'Acting Under Secretary of Commerce 
I for Intellectual Property 
and Acting Director of the U.S. 
iPatent and Trademark Office 



■0 



Pfepsd type a plus pign {+) inside (his box 

PTOJSB/16 (02-01) 
Approved for use through 10731/2002. OMB 0551-0032 
Patent and Trademark Office; US. DEPARTMENT OF COMMERCE 
Under the Paperwork Reduction Act of 1995. no persons am required to respond to a collection of information unless it displays a valid OMB control number 

PROVISIONAL APPLICATION FOR PATENT COVER SHEET 
This is a request for filing a PROVISIONAL APPLICATION FOR PATENT under 37 CFB 1 .53 (c). 
| Expr S3 Mall Label No. EL995079232US I 



O 
H 
CL 



t — 

OtCD 
<J»VO 

l> 



INVENTOR(S) 



Given Name (first and middle [if any}) 



Family Name or Surname 



Residence 
(City and etthei Slate or Foreign Counlry) 



Jill MacDonald 
Cristina 



Boyce 
GomUa 



ManaJapan, New Jersey 
Princeton, New Jersey 



E Additional Inventors are being named on the 1 separately numbered sheets attached hereto 



TITLE OF THE INVENTION (280 characters max) 



VIDEO COMFORT NOISE 



Direct all correspondence to: 
□ Customer Number 
OR 



CORRESPONDENCE ADDRESS 



Type Customer Number here 



Place Customer Number 
Bar Code Label here 



Firm or 

Individual Name 



JOSEPH S. TRIPOLI, THOMSON LICENSING INC. 



Address 



PATENT OPERATIONS. 



Address 



P. O. BOX 5312 



City 



PRINCETON 



State 



NJ 



ZIP 



08543-5312 



Country 



USA 



Telephone I 609-734-6834 



Fax 



609-734-6888 



ENCLOSED APPLICATION PARTS (check all that apply) 



I3 Specification Number of Pages I 5 I 
Drawing(s) Number of Sheets 



□ CD(s), Number 
n Other (specify) 



E] Application Data Sheet. See 37 CFR 1.76 



METHOD OF PAYMENT OF FILING FEES FOR THIS PROVISIONAL APPLICATION FOR PATENT (check one) 
Q Applicant claims small entity status. See 37 CFR 1.27. 
CI A check or money order is enclosed to cover the filing fees 

I3 The Commissioner is hereby authorized to charge filing 

fees or credit any overpayment to Deposit Account Number 
O Payment by credit card. Form PTO-2038 is attached. 



FILING FEE 
AMOUNT ($) 



Q7-QS3Z 



16Q 



The invention was made by an agency of the United States Government or under a contract with an agency of 
the United States Government. 

HNo. 

D Yes, the name of the U.S. Goveinmfent agency arm the Government contract number are: . 




Respectfully submitted, 
SIGNATURE 



TYPED or PRINTED NAME 
TELEPHONE 609 734-6820 



Date 



8/20/03 



REGISTRATION NO. 
(if appropriate) 

Docket Number 



28,234 



PU030249 



USE ONLY FOR FILING A PROVISIONAL APPLICATION FOR PATENT 

This coBectfon of information is required by 37 CFR 1.51. The information is used by the public to file (and by the PTO to process) a provisional application 
Confidentiality Is governed by 35 U.S.C. 1 22 and 37 CFR \ .14. This collection Is estimated to take 8 hours to complete, including gathering, preparing, and submitting 
the complete provisional application to the PTO. Time will vary depending upon the individual case. Any comments on the amount cf time you require to complete this 
form and/or suggestions for reducing mis burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office. U.S. Department of Commerce. 
Washington. D.C.. 20231. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO Box Provisional Application, Assistant Commissioner tor 
Patents, Washington. D.C. 20231. 



PROVISIONAL APPLICATION COVER SHEET 

Additional Page 

PT0/SB/16 (02-01) 
Approved lor use through 10/3172002. OUB 0651-0032 
Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE 
Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a vafid OMS contra) number. 



Docket Number 


Type a plus sign (+) 
PU030249 inside this box — > + 


INVENTOR(SVAPPLICANT(S) 


Given Name {first and middle fit any]) 


Family or Surname 


Residence 
(City and either State or Foreign Country) 


i Jeffrey Allen 


Cooper 


Rocky Hill, New Jersey 



Number 2 of 2 



WARNING: Information on this form may become public. Credit card information should not 
be included on this form. Provide credit card information and authorization on PTO-2038. 



PTO/SB/17(01-03) 

% Approved for use through 04/30/2003. OMB 0851 -0032 

U.S. Palem and Trademark Office: US. DEPARTMENT OF COMMERCE 
Under the Paperwork Reduction Ad d 1995, no persons are required to respond to a collection o1 information unless it displays a valid OMB control number 



r 



FEE TRANSMITTAL 
for FY 2003 

Effective OWi/2003. Patent fees are subject to annual revision. 



□ Applicant claims small entity status. See 37 CFR 1.27 



Application Number 



Filing Date 



First Named Inventor 



Examinar Name 



Group /Art Unit 



Complete if Known 



jjjj MacDonaid Boyce 



VTOTAL AMOUNT OF PAYMENT 



i$L 



160 



Attorney Docket No. PU03O249 



METHOD OF PAYMENT (check ail that apply) 



FEE CALCULATION (continued) 



□ Check □ Credit card □ Money □ Other □ None 
Order 

E9 Deposit Account 



3. ADDITIONAL FEES 



Deposit 
Account 



07-0832 



Deposit 
Account 
Name 



Thomson Licensing Inc. 



Trie Commissioner is authorized to: (check ai) that apply) 
S Charge fee(s) indicated below 0 Credit any overpayments 
0 Charge any additional fee(s) during the pendency of this application 
□ Charge fee(s) indicated below, except for the filing fee 
to the above-identified deposit account 



J deposit e 

FEE CALCULATION 



BASIC FILING FEE 



Larae Errtftv 


Small Errtftv 






Fee Fee 


Fee 


Fee 


Fee Descrlotion 




Code ($) 


Code 


<$> 


Fee Paid 


1001 750 


2001 


375 


Utflity filing fee 




1002 330 


2002 


165 


Design filing fee 




1003 520 


2003 


260 


Plant filing fee 




1004 750 


2004 


375 


Reissue fifing fee 




1005 160 


2005 


80 


Provisional filling fee 


160 



SUBTOTAL (1) 



2. EXTRA CLAIM FEES 



Total Claims 



Claims 

Multiple 
Dependent 



•20* 



Extra 
Claims 



Fee from 
bejpw 



Fee 

Paid 



Larae Entttv 




Fee 


Fee 


Fee Fee 


Code 


($) 


Code (S) 


1202 


16 


2202 9 


1201 


84 


2201 42 


1203 


280 


2203 140 


1204 


04 


2204 42 


1205 


18 


2205 9 



FeeOes^rfptign, 

Claims in excess of 20 
Independent claims in excess of 3 
Multiple dependent claim, H not paid 
" Reissue independent claims over 
original patent 

" Reissue claims In excess of 20 and 
over original patent 



Fee 


Fee 


Fee 


Fee 


Code 


(» 


Code 


($) 


1051 


130 


2051 


65 


1052 


50 


2052 


25 


1053 


130 


1053 


130 


1812 


2.520 


1612 


2,520 


1804 


920* 


1604 


920* 


1805 


1.840* 


1805 


1,840* 


1251 


110 


2251 


55 


12S2 


410 


2252 


205 


1253 


930 


2253 


465 


1254 


1,450 


2254 


725 


1255 


1.970 


2255 


985 


1401 


320 


2401 


160 


1402 


320 


2402 


160 


1403 


280 


2403 


140 


1451 


1,510 


1451 


1.510 


1452 


110 


2452 


55 


1453 


1,300 


2453 


650 


1501 


1.300 


2501 


650 


1502 


470 


2502 


235 


1503 


630 


2503 


315 


1460 


130 


1460 


130 


1807 


50 


1807 


50 


1606 


160 


1606 


180 


8021 


40 


8021 


40 


1809 


750 


2809 


375 


1810 


750 


2810 


375 


1801 


750 


2801 


375 


1802 


900 


1802 


800 



Fee Description 

Surcharge - late filing fee or oath 

Surcharge - late provisional filing fee 

or cover sheet. 

Non- English specification 

For filing a request for reexamination 

Requesting plication of SIR prior to 

Examiner action 

Requesting publication of SIR after 
Examiner action 

Extension for reply within first month 
Extension tor reply within second 



Fee Paid 



Extension for re pry within third month 
Extension for reply within fourth 
month 

Extension for reply within fifth month 
Notice of Appeal 

Fifing a brief in support of an appeal 
Request for oral hearing 
Petition to institute a public use 



Petition to revive - unavoidable 
Petition to revive - unintentional 
Utility issue fee (or reissue) 
Design issue fee 
Plant issue fee 

Petitions to the Commissioner 

Processing fee under 37 CFR 1 .1 7 (q) 

Submission of Information Disclosure 
Stmt 

Recording each patent assignment 
per property (times number of 
properties) 

Fifing a submission after final rejection 
(37 CFR § 1.129(a)) 
For each adortionaJ invention to be 
examined 137 CFR § 1 .129(b)) 

2801 375 Request tor Continued Examination (RCE) 

Request lor expedited examination 
of a design application 



SUBTOTAL (2) ($)0 



Other fee (specify) _ 



'Reduced by Basic Rling Fee Paid 



SUBTOTAL (3) 



"or number previously paid, if greater; For Reissues, see above 



(S)0 




WAgtaNG: Information en this form may become public. Credit card Information should not be 
Included on this form. Provide credit card information and authorization on PTO-2038. 
This coOection of information is required by 37 CFR 1.17 and 1.27. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to 
process) an apc*cation. Confajemiafity is governed by 35 U.S.C. 122 and 37 CFR 1.14. This coflecticn is estimated to take 12 minutes to complete, including gathering, 
preparing, and submitting the completed application lorm to the USPTO. Time wiQ vary depending upon the individual case. Any comments on the amount of time you require to 
complete this form and/or suggestions for reducing this burden, should be sent to the Croat MormatJon Officer, U.S. Patent and Trademark Office. U.S. Department of 
Commerce. Washington, DC 2023V DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, Washington, DC 20231. 
H you need assistance in completing this form. caO 1-80&PTO-9199 (1-800-78$~9199) and select option 2. 



Cf pjuui. Iw^teJL ^»JUJi ho e^HSO'Ttazj.US 



I A. Brief summary df^tfie inventi n 



: i j * zrr* 



Video compression at low bitrates tends to cause visible artifacts that are noticeable 
to a viewer. The human visual system causes some types of artifacts to be more 
noticeable and objectionable than others. Addition of random noise can reduce the 
noticeability of compression artifacts, but large frame-to-frame differences created by 
random noise addition are themselves noticeable and objectionable. In this invention, 
temporal correlation is used in the creation of the additive comfort noise pattern to 
reduce the amount of frame-to-frame differences. 




H.264, JVT, MPEG AVC 
dither 

comfort noise 



Video compression at low bitrates tends to cause visible artifacts that are noticeable 
to a viewer. The human visual system causes some types of artifacts to be more 
noticeable and objectionable than others. Blockiness and structured noise patterns are 
common artifacts in block-based image and video compression techniques. Addition 
of random noise can reduce the noticeability of compression artifacts, but large 
frame-to-frame differences created by random noise addition are themselves 
noticeable and objectionable. 




It is well known that addition of a dither signal can reduce human sensitivity to image 
artifacts, for example to hid contouring and blocking artifacts. In [1], a random noise 
dither based on film grain is added to an image to hide block effects, with the rationale 
"random error is more forgiving than the structure, or correlated error." 

It has also been previously proposed that a dither signal be added to video sequences to 
hide compression artifacts. In [2], it was proposed that a random noise dither be added 
in the video encoding and decoding process in the in loop deblocking filter for the JVT 
video coding standard [3]. The amount of dither to be added depended on the position of 
a pixel with respect to a block edge. 

The term "comfort noise" is used in audio compression to indicate noise pattern 
generated at the receiver end, to avoid total silence that is uncomfortable to a listener. 
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It was proposed in [4] that random noise be added as a postprocess to video decoding, for 
use as comfort noise. The amount of noise added depends on the quantization parameter 
and the amount of noise added to spatially neighboring pixels. 



[1] S. Yang, Y. Hu, "Blocking Effect Removal Using Regularization and Dithering", 
ICIP 1998, p. 415-419. 

[2] G. Conkin, N. Gokhale, JVT-C-56, "Dithering 5-tap Filter for Inloop 
Deblocking," ftp://ftp.imtc-files.org/jvt-experts/2(K)2_-5„Fairfax/JVT-C056.doc 



[3] T. Wiegand, JVT Study of Final Committee Draft, Dec 5-13, 2002, ftp://ftp.imtc- 
files.org/jvt-experts/2002 12 Awaji/JVT-F 100dlncm.zip 



[4] G. Bjontegaard, "Addition of 'comfort noise' as post processing", ITU-T SG 16 
Q15-B-15. 



A dither signal containing random noise is added to the pictures in a video sequence, after 
video compression decoding, to improve the subjective video quality. Figure 1 shows a 
receiver end system, which includes a video decoder and a comfort noise generator. The 
generated noise is added to the decoded pictures prior to display. The addition of the 
comfort noise occurs after reference picture storage, as the reference pictures must be 
unchanged in order to properly decode the following pictures. Figure 2 shows another 
receiver end system, in which the noise generator uses as input the decoded picture, and 
other bitstream information from the video decoder. Figure 2 is compatible with the 
comfort noise generation described in [4], in which the quantization parameter is the 
Bitstream information passed from the video decoder to the noise generator. 



r . » 



Vld 
D coder 



Decoded 
pictures 



Reference 
picture 
stores 



Noise 
Generator 



To 
Display 



Figure 1. 



Video 


Decoded 
pictures 




Noise 






Decoder 


Bit stream 




Generator 











To 
Display 



Reference 
picture 



Figure 2. 



In this invention, the determination of the magnitude of the noise signal to add is based 
on any of several different factors. Let N(k, x, y) be the added pixel noise signal, P (k, 
x, y) be the decoded pixel, and D (k, x, y) be the displayed pixel (x, y) of the k-th picture 
in the video sequence, respectively. The displayed pixel of the picture is the decoded 
pixel plus the noise signal, 

D(k,x, y) = P(k, x,y) + N(k, x, y) 

The visual impact of adding a noise signal to the video sequence, rather than just to a 
single image, is considered in the determination of the magnitude of the noise signal. 
The magnitude of additive noise signal for a pixel in a picture is correlated to the value of 
the additive noise signal of the pixels in the previously displayed picture, e.g., the noise 
signals are temporally correlated. In an alternative embodiment of the current invention, 
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the temporal correlation is based on the previously decoded picture, rather than the 
previously displayed picture. 

In an embodiment of the current invention, the added noise signal using temporal 
correlation, with correlation factor a, 0 < a < 1, is computed as 

N(k, x, y) = a N(k-1 , x, y) + (1 - a) R(k, x, y) (Eq. 1) 

The random number R(k, x, y) may be generated using any type of random number 
distribution, for example a Gaussian or Laplacian distribution. The random number 
generator may be implemented by means of a lookup table. R(k, x, y) may also include 
spatial correlation, such as that used for example in film grain noise generation. 

Because a is fractional, to avoid divisions, Eq. 1 can be simplified, 
N(k, x, y) = (a * N(k-1 , x, y) + (2 b - a) R(k, x, y) + 2 W> » b (Eq 2) 

Figure 3 shows a receiver-end system that includes a noise generator compatible with 
Eq. 1 or Eq. 2. In Figure 3, the noise signal N(k, x, y) for the k-th picture is stored in the 
noise picture store. 
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Figure 3. 



The system of Figure 3 requires more memory bandwidth than the system of Figure 1, 
because for each displayed picture the previous displayed picture's noise signal picture 
must be accessed and the current picture's noise signal picture must be stored. 



The system of Eq. 1 and Figure 3 is implementation is an instantiation of a temporal IIR 
filter. The IIR filter may be generalized by using more filter taps. IIR filters can also 
generally be approximated using higher order FIR filters, using as many taps, t, as 
desired. 

N(k,x, y) = £ a ' +1 x > y) + a "(I- a ) R ( k -'> x > y) ^ 3 ) 

An FIR filter approach can be implemented using the system of Figure 2. Only the 
previous random numbers R, rather than the previous noise, N, is used in an FIR filter 
approach, so memory bandwidth is reduced. 

Possible claims: 

1 . Add temporally correlated noise as a postprocess following video decoder 

2. Claim 1 using a correlation factor alpha 

3. Claim 1 with FIR filter implementation 
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